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OFFICE OF THE SECRETARY OF DEFENSE 
WASHINGTON, D C. 20301-3140 


3 0 JUN 1994 


MEMORANDUM FOR UNDER SECRETARY OF DEFENSE (ACQUISITION & 

TECHNOLOGY) 

SUBJECT: Report of the Defense Science Board Task Force on 

Acquiring Defense Software Commercially 

I am pleased to forward the report of the Defense Science 
Board Task Force on Acquiring Defense Software Commercially. The 
Task Force, co-chaired by Dr. George H. Heilmeier and Dr. Larry 
Druffel, was chartered to determine the conditions under which 
defense software could be procured using commercial practices and 
to develop a strategy for procurement that incorporates such 
practices. 

In its investigation into applying commercial practice to 
DoD software procurement, the Task Force reviewed a broad 
spectrum of interrelated elements from software program 
management policy and DoD software acquisition practices to DoD's 
investment in the software technology base. I wholeheartedly 
concur with the group's key finding that DoD must develop a more 
coordinated approach to the oversight of its diverse software 
capabilities and programs. The Task Force's report provides an 
outstanding point from which to begin addressing revamping DoD's 
software procurement practices. 
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BOARD 




Paul G. Kaminski 
Chairman 


OFFICE OF THE SECRETARY OF DEFENSE 
WASHINGTON. D.C. 20301 -3140 


3 0 JUN 1994 

DEFENSE SCIENCE 
BOARD 

Dr. Paul Kaminski 

Chairman, Defense Science Board 

Office of the Undersecretary for Acquisition and Technology 
The Pentagon 
Washington, D.C. 



Dear Dr. Kaminski: 

Enclosed is the final report of the Defense Science Board Task Force on Acquiring Defense 
Software Commercially. We were tasked to determine the conditions under which procurement of 
defense software can use commercial practices and to define needed changes to permit such use. 
In the attached report, we make specific recommendations with regard to DoD process credibility, 
software program management, required expertise of DOD personnel using modern software 
practices, use and integration of commercial off-the-shelf software, DoD software acquisition 
practices, use of software architectures by DoD as a management tool, and DoD’s investment in the 
software technology base. 

Based on our review, we concluded that issues associated with defense software are 
applicable across the wide spectrum of DoD software intensive systems. However, to reap 
maximum benefit from improvements related to the myriad aspects associated with software, the 
DoD requires a more coordinated approach to the oversight of its diverse software capabilities and 
programs. 

To provide such Department-wide oversight and to facilitate implementation of the other 
specific recommendations contained within this report, the Task Force recommends that the 
Secretary of Defense assign to the Under Secretary of Defense (Acquisition and Technology) the 
responsibility for DoD-wide software technology, policy, practices and acquisition. This 
centralized approach will best serve the Department for all software intensive systems and 
programs. To support the implementation of this recommendation and to ensure that the key 
stakeholders are participants in the process, the Task Force further recommends the formation of 
an Executive Council consisting of the appropriate principals from OSD, the Services, and the 
Defense Agencies. 

We thank all of the members and government advisors of this Task Force for their 
dedicated efforts and significant contributions to this study. 
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Executive Summary 


The Defense Science Board Task Force on Acquiring Defense Software Commercially recognizes 
that DoD systems are becoming increasingly dependent on the use of software as the mechanism for 
implementing operational capabilities. To adapt to changing military and national security situations, DoD 
is more dependent than ever on its ability to modify mission software rapidly, often in near real-time. 
However, software remains the schedule and cost driver for the development and maintenance of many 
important defense systems. 

In its review of current DoD and commercial software acquisition practices, the Task Force found 
notable differences, as evidenced in Appendix C. There are, however, many similarities between the 
various categories of DoD and commercial software systems. Although there are indications that 
commercial development efforts have achieved better predictability and lower costs, the Task Force noted a 
significant lack of credible, quantitative data to substantiate this assessment. 

In general, the Task Force concluded that DoD's investment in software requires greater DoD-wide 
management control and oversight in the coming years if the Department is to exploit the use of 
commercial software acquisition practices fully, as well as rapid advances in software technology. The 
following is a summary of selected findings and recommendations toward that end. 

Process Credibility: Current DoD practice is not compatible with commercial business 

practices. DoD should work to make necessary changes to acquisition regulations such as: 

• Having program managers manage 3 of 3 (price/schedule/functionality) but only constrain 2 of 3 

• Defining successful performance on contracts as delivering a solution (with predictable price, 
schedule and functionality) not adherence to government processes, procedures and specifications 

• Not requiring c-level specifications for software projects developed in Ada 

• Establishing mechanisms to allow both current ability to perform as well as past performance as 
key factors in source selection 

• Encouraging offerors to demonstrate as much functionality as possible as part of bid without 
eliminating domain knowledgeable competition 

DoD Program Management: DoD program management approaches discourage the use of 

commercial practices. Program managers lack incentives to tailor procedures to fit individual program 
needs or to develop “corporate” solutions (e.g., employ common architecture or common software 
components). DoD should establish and implement overarching software life cycle guidelines more 
conducive to the use of commercial practices and products, such as: 

• Defining software architectures to enable rapid changes and reuse 

• Facilitating early system engineering and iterative development 

• Participating in development of commercial and international standards 

• Allowing the fielding of software directly from test beds with user consent 

• Requiring program managers to stay with programs at least through beta testing to maintain 
continuity of understanding of original nuances in requirements 
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DOD Personnel: There is currently a shortage of sufficiently qualified software personnel at all 
levels within the Department. DoD should establish a Department-wide software program management 
education and training initiative that includes: changing courses for PMs to reflect best commercial 
practices and other recommendations of this task force and providing for changes to reflect the dynamics 
of the software industry; rotating government and contractor personnel between PM and developer 
organizations to build understanding and trust; encouraging use of IPA's from industry; and integrating 
software-qualified personnel into senior DoD acquisition staff. 

Use and Integration of Commercial Off-the-Shelf (COTS) Software: DoD has not 

fully identified the pros and cons associated with the use of COTS software and, as a result, has not 
determined when and how best to use COTS software. To facilitate this process, DoD should require 
trade studies and analysis of the use of COTS software in DoD’s software acquisition process where 
appropriate. Further, DoD should establish “customer friendly,” application-specific information 
technology “component stores” to enable program managers to assemble systems rather than develop them 
through use of reusable, prequalified components. DoD should also increase technology base funding for 
security audit tools for systems employing COTS software and should capitalize on innovative cost- 
effective techniques for acquiring and using COTS software products, such as the use of enterprise 
licenses. 

Software Architecture: Software architecture was emphasized by the Task Force as a means 
for achieving important ends. There is currently little emphasis on architecture in DoD software programs 
or regulations. As a result, DoD is not benefiting from architecture as a key tool for evolutionary 
development and for early (and frequent) involvement of users with functional capability and facilitating 
reuse, requirements changes with minimum cost and schedule, and “product line” management. DoD 
should require vendors to propose, manage and control the architecture and should establish an early 
architecture deliverable in all developments. 

Software Technology Base: The current DoD software technology base investment does not 
adequately take advantage of commercial R&D. Further, software technology transfer (both internal and 
external) is not receiving adequate emphasis within DoD. DoD should provide for the evolution of the 
DoD Software Technology Strategy to align with emerging commercial technology and practices. 

Overarching Recommendations: To facilitate implementation of the many recommendations 
contained in this report, the Task Force concluded that DoD's investments in software require greater 
management control and DoD-wide oversight. To this end, the Task Force recommends that the Secretary 
of Defense (SECDEF) assign the Under Secretary of Defense (Acquisition and Technology) (USD (A&T)) 
the responsibility for DoD-wide software technology, policy, practices and acquisition. In carrying out 
this responsibility, the Task Force recommends that the USD (A&T) consider forming an executive 
council with the Director, Defense Research and Engineering (DDR&E), the Assistant Secretary of 
Defense (Command, Control, Communications and Intelligence) (ASD (C3I)) and appropriate 
representatives from the Services and Defense Agencies as members. Further, USD (A&T) should 
provide for a supporting “process action team” to assist in implementation of the recommendations of this 
Task Force study. 


Defense Science Board Task Force 
on Acquiring Defense Software Commercially 


1.0 TASK FORCE OVERVIEW 

1.1 TERMS OF REFERENCE 



Terms of Reference 



Objectives: 

• Determine: 

- Conditions Under which Procurement of Defense Software Can Use Commercial Practices 

- Changes Required to Permit Such Use 

• Develop Strategy that Incorporates Such Practices 

- Not Constrained by Existing DoD Standards 

- Viewed as Coexisting Alternative Strategy 

- Includes DoD Use of Commercial Software Products 

• Compare Proposed Strategy with Current DoD Strategy, Indicating 
Circumstances Where Each is Most Beneficial 

Scope: 

• All Software Intensive Systems 

• All Stages of the Software Life-Cycle 

V J 

Appendix A provides the Terms of Reference by which the Task Force was established. The 
objectives and scope of this Task Force are outlined above. At the first meeting of the Task Force, the 
sponsor of the study (the Director, Defense Research and Engineering) requested that the Task Force 
provide a strategy that: was sensible, pragmatic, and unconstrained by current DoD acquisition practices; 
was based on evaluation of mechanisms for integrating defense software efforts with commercial software 
efforts; did not require legislative relief; and addressed the full spectrum of DoD software applications. 
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1.2 CAVEATS 



Caveats 



• Relied on Inputs from DoD and Industry Experts 

• Provided No Detailed, Quantitative Assessment or 
Evaluation of Individual Topics 

• Did Not Address: 

- Success or Failure of Ada 

- Recommended Software Development Approach for Specific 
Programs 

- Specific COTS Products for DoD to Exploit 

- Trade-Offs Between Hardware and Software 


V J 

The Task Force relied heavily on inputs from a variety of DoD and industry experts regarding a 
wide range of topics related to defense software technology, policy, practices and acquisition. Appendix 
B lists the many briefings that were provided. Although the Task Force chose not to provide detailed, 
quantitative assessments or evaluation of individual topics, it used the information associated with these 
topics in the formulation of findings and recommendations. The areas not addressed by the Task Force, 
while important in and of themselves, were determined not to be directly germane to the development of an 
overall defense software strategy. 
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1.3 TOPICS ADDRESSED 



In its efforts to assess the appropriateness of DoD use of commercial practices, the Task Force 
addressed software management, contracting, and technical issues. The above viewgraph lists the primary 
topics considered within each of these categories. 
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1.4 MEMBERSHIP 



Members 



Co-Chairmen: 

• Dr. George H. Heilmeier 

• Dr. Larry Druffel 

Members: 

• Dr. Jacques Gansler 

• Mr. Jack Hancock 

• Mr. Patrick Hillier 

• Mr. Arthur E. Johnson 

• Dr. Bruce Johnson 

• Mr. Alan McLaughlin 
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Bellcore 
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• Mr. William Mounts 
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ODASD(IM) 
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Task Force members represented a valued cross section of software expertise within both the DoD 
and commercial sectors. The government advisors represented senior executives (including the three 
Service Software Executive Officials) from the major software management organizations within the 
Department. 
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2.0 


CURRENT POD AND COMMERCIAL SOFTWARE ACQUISITION PRACTICES 
2.1 BACKGROUND 



Background: Previous Studies 




DSB Summer Study on Technology Base (1981) 

Joint Service Task Force on Software Problems (1982) 

AF SAB High Cost and Risk of Mission Critical Software (1983) 

CODSIA Report on DoD Management of Mission-Critical Computer Resources (1984) 
DSB Task Force on Military Software (1987) 

Ada Board Response to DSB Task Force (1988) 

Summer Report on Defense-Wide Audit of Support for Tactical Software (1988) 
Workshop on Executive Software Issues (1988) 

Workshop on Military Software (1988) 

Army Materiel Command Study (1989) 

Software Technology Development and Deployment Plan for DoD Technology Base 
(1989) 

AF SAB Adapting Software Development Policies to Modem Technology (1989) 

Draft DoD Software Master Plan (1990) 

Draft DoD Software Technology Strategy (1991) 

AF SAB Study on Information Architecture (1993) 

Study on Military Standards Impacts on the Acquisition Process (1993) 

Draft Software Action Plan Working Group Report (1993) 

Evolutionary Acquisition Study, AFCEA (June 1993) 




As a point of departure, the Task Force noted a number of previous studies addressing issues 
related to the defense software technology, policy, practices and acquisition. Despite the increased 
emphasis given to software issues by the DoD (as evidenced by the above list), the majority of the 
recommendations resulting from these studies have not been implemented. 
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Background: Impediments to Change 



• DoD Software Management 

No Single Operative Mechanism Exists to Implement Change 
Three Separate Domains: MIS, C3I, Embedded 
Two Separate Acquisition Management Structures 
Bureaucratic "Turf" Inhibits Real Reform 

• Acquisition Process 

5000 and 8000-Series Developments Typically Employ "Waterfall" 

Approach; Not Incremental/Spiral Approach 

• Culture 

Systems Are Stove Pipe — My System, My Program (PM is King) 

DoD Acquisition Training Reinforces the Wrong Approaches 

• Procurement 

The Contracting Process Inhibits Creativity and Investment by 

Contractors; Limits Options 

Interpretation of Competition in Contracting Act 

V J 

To ensure that its recommendations could be readily implemented by the DoD, the Task Force 
identified the primary reasons why recommendations from previous studies had not been acted upon. The 
Task Force then formulated its recommendations to appropriately address these impediments. 
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Why Is This Study Important? 



• DoD Military Capability Increasingly Software Dependent 

• Software One of Few DoD Budget Areas That is Projected to Grow 

- Software Has Become the Overall Defense System Schedule and Cost Driver 

» Declining DoD Budget - Affordability a Major Concern 
» Escalating Costs of Software Development and Life Cycle 

- Significant Level of DoD Resources To Be Spent on Information Technology: 

» FY92-FY94: ~$10 Billion/Year* 

» EIA Forecast for FY95 - FY98: ~$10 Billion/Year 
» In-House vs. Contracted Out: 30% In; 70% Out 

- Will Require Greater Management Control 

• Rich Commercial Base to Tap; Many Opportunities 

- Custom Software - Acquisition Methodologies 

- Off-the-Shelf Products - Approach to Requirements Determination 

• Functionality and Flexibility More Embedded in Software than Hardware 

• Study is Timely Because DoD Leadership is Focused on Acquisition 
Reform 

- Something May Actually Get Done 




Given its increasing reliance upon software as the mechanism for implementing system 
capabilities, coupled with the rising costs associated with software development and maintenance, DoD 
must take action now to address the issues associated with software acquisition. The commercial sector 
provides numerous opportunities upon which the DoD could readily capitalize. The time is ripe for 
assertive DoD action, particularly since the current leadership is so strongly focused on acquisition reform. 
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The Software Domain 
is Very Large 
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COST TO DEVELOP 


The software domain reviewed by the Task Force encompasses a wide variety of DoD systems, 
ranging from software tools to large embedded real-time systems. The associated cost, complexity, and 
time required to develop these systems vary widely, both for commercial and military applications. 
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Background: 

DoD Software Acquisition Management 



• Two Different Sets of Software Policies/Rules Managed by 
Two Different Organizations 

- USD (A&T) for Software Embedded in Weapons/Systems 

- ASD (C3I) for C3I Software and MIS Applications 

• Majority of Software Issues are Applicable to All DoD 
Systems 

• Growing Similarity Between DoD and Commercial Software 
Developments Across the Various Types of DoD Software 

— Modem Software Tending to Blur Distinction Between DoD and 
Commercial Applications 


Central DoD Leadership Needed More than Ever 

— Major Revisions Needed in DoD Software Policies/Rules and 
Management Across All DoD Applications in Order to Meet SECDEF's 
Acquisition Reform Goals 

- Interpretation and Implementation of DoD-Wide Policies/Rules by 
Management is a Central Issue 


Today, the management of DoD software acquisitions is quite complex. There are two sets of 
software policies/rules managed by two different organizations 

• USD (A&T) for software embedded in weapons/systems 

• ASD (C3I) for C3I software and MIS applications 

There currently exists within the DoD a dichotomous organizational structure for the management 
of DoD software intensive systems. The Under Secretary of Defense (Acquisition and Technology) is 
responsible for weapons systems; the Assistant Secretary of Defense (Command, Control, 
Communications, and Intelligence (C3I)) is responsible for C3I systems and Management Information 
Systems (MIS). However, as technology has evolved over the years, the issues associated with the use of 
software in all of these systems have become essentially the same. The issuance of different software- 
related policies and regulations that are oriented according to the type of system is no longer appropriate. 
If the DoD is to adequately address this fundamental issue, central focused management is required. 
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2.2 COMPARISON OF DEFENSE AND COMMERCIAL SOFTWARE 



Comparison of 

Defense and Commercial Software Needs 



• Defense and Commercial Software Applications Needs Are Merging 



Breadth of 
Application 

Typical Size 
(SLOC) 

Upgrade/ 

Change 

Sensitivity to 
Errors 

Real Time Applications 

Unique 

400,000 

Complex; 

Inflexible 

High 

- DoD Real Time Software 

- Commrcial Custom Software Integrated 
Into Real-Time Operations 

Unique 

400,000 


High 


Moving Toward 
Wide 

200,000 

Complex; Flexible 

Moderate 

- C3I 

- Commercial Custom Software Integrated 
Into Business Operations 

Moving Toward 
Wide 

200,000 

mSBEBSm 


MIS Svstems 

Very Wide 

200,000 

Moving Toward 
Commercial 

Low 

- DoD Automated Information Systems 

- Commercial Automated Information 
Systems 

Very Wide 

200,000 

Exploits Object 
Oriented 
Technology 

Low 

Reusable Components/ Products 

Very Wide 

50,000 

Tied to Weapon 
System Cycle 

Low to High 
(Depending on Use) 

- DoD Component Stores 

- Commercial Shrink Wrapped 

Very Wide 

50,000 

Tied to 

Commercial Cycle 

Low 


DoD and commercial applications for software are different in some ways but very similar in 
others. The above table highlights these similarities and differences for real-time applications, command 
and control applications, MIS systems and reusable components and products. This apparent disparity in 
the classification of software between DoD and commercial vendors increases the complexity of DoD's 
management task. Companies (people, methods, and organizations) are usually specialized to one or more 
commercial type systems, not to DoD type systems. Defense and commercial software applications needs 
are merging, providing the potential that DoD can exploit commercial capabilities more effectively over 
time. Major revisions are needed in DoD software management across all DoD applications in order for 
DoD to capitalize on the evolving commercial base. 
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Process Comparison Summary 





Custom 

Custom 

Commercial 

Civilian/ 

Commercial 

Process Attribute 

DoD 

Commercial 

Product 

Result 


Problem Definition 

: Owned by Originator 

Shared (Includes 
Market Analysis) 

Marketplace 

More Practical 
Requirements 

Process Definition 

By Spec, Rigid 

Intemal/EvoMng 

Intemal/Evolving 

Reduced Cost/ 
Schedule 

Flexibility 

Very limited 

Not Constrained 

Essentially 

Unlimited 

Process Improvement 
Ease, Enabling 
Technologies 

Mllestones/Reviews 
{Duration, Formality) 

tong, Formal, II 
Heavy Document 

Short Informal 
Incremental 
Document 

Short, Informal 

Reduced Cost/ 
Schedule 



EMM! 

Beta 

Less Requirements 
Churning. Reduced 
Cost/Schedule 

Process Monitoring by 
Customer 

Heavy 

Some 


Reduced Cost / 
Schedule 

Customer Acceptance 
Process 

Rigorous 

Simple 

Marketplace 

Reduced Cost/ 
Schedule 

Inspection/Testing 

, Rigorous /Formal 

Rigorous/Formal 

Rigorous/Formet 

Similar Product 
Quality 

Subcontracting 

By Spec 

Brief Product Spec 
ISO Possible 

Brief Product Spec 
ISO Possible 

Quallty/DependabiHty 
of Subcontractor May 
Be Less 

\mmsa 

. , V Implicitly 
Discouraged 

Extensive 

Extensive 

Reduced Cost/ 
Schedule Higher 
Quality 


Source: IBM Federal Systems 



As is evident from the process comparison summary above, there are not only differences between 
commercial and defense applications, but also in the process used by each to develop, acquire and support 
complex software systems. Defense and other federal acquisition regulations and detailed specifications 
require a much more complex set of deliverables within the context of DoD’s contracting process. 
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Comparison of DoD and Commercial 
Software Acquisition Practices 

• Requirements Definition 

- Commercial 

- More flexible and open between users and supplier 

- Based on strategic plan and usually cost/schedule driven 

- More willing to adjust requirements based on availability of off-the-shelf 
products 

- Evolves capability 

• Vendor Selection 

- Commercial 

- Much more flexible; no requirement for fairness or to maintain the public trust 

- Encourages vendors to offer best solution, not meet 100% of requirements 

- Accommodate teaming and long-term relationships 

• Development Process 

- Commercial 

- More flexible; product improvements anticipated 

- Team approach with bias toward reuse and tailoring of existing systems 

- Multi-year acquisitions not re-justified each year. 


In essence, the Task Force found major differences between DoD and commercial software 
acquisition practices, as outlined above and on the next page. 
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Comparison ofDoD and Commercial 
Software Acquisition Practices (Cont.) 



• Business Practices 

- Commercial practice more flexible with greater incentives 


• Integration Testing and Delivery 

- Commercial provides integration and functional testing according to need 

- DoD uses separate test agency with added time and complexity 

- Absence of beta testing within DOD increases costs 


• Rights in Data 

- Commercial more flexible, especially regarding resales 


• Maintenance 

- Commercial: Maintenance considered and integrated with development 

- DoD: Maintenance not major factor in development process 




Appendix C provides a more complete comparison of DoD and commercial software acquisition 
practices as developed by this Task Force. 
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Assessment of Current DoD 
Software Acquisition Approach 



• Strengths 

- Highly Structured Process Tied to Individual System 
Developments 

- Tightly Defined Requirements 

- Produces High Quality Product for Mission Critical Systems 
That Demand Extremely Low Failure Rates (e.g.. Flight 
Controls for Man-Rated Platforms) 




The strengths of the current DoD approach to software development, acquisition and operation are 
summarized above. The Task Force found that the highly structured DoD process has, in fact, provided a 
high quality software product, in most cases. 
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Assessment of Current DoD 
Software Acquisition Approach 




Weaknesses 

- High Life Cycle Cost in Time and Dollars 

- Software Development Cycle Tied to Weapon System Developments 

» Incredibly Long , Typically 13 to 15 Years from Concept to Fielding 

- Encourages Excessive Acquisition Agent Involvement in Design 
Detail and Process 

- Based on Mistrust vs. Mutual Trust 

» Excessive Documentation 
» Excessive Formal Review 
» Excessive Testing of Non-Critical Systems 

» Poor Communication Between Vendor, Acquisition Agent and User 

- No Requirements Addressing Cost and Schedule 

- Traditional Approach Used: Design it All and Then Build it 

- Little or No Requirements Relaxation for High Cost Items 

- Inadequate Beta Testing in Early Phase 

- Little Focus on Designing in Reusability 



However, there are a number of weaknesses associated with the current defense approach, as 
summarized above. Many of these weaknesses derive from the need for a fair and open procurement 
process and the necessity to prove that public dollars are wisely spent. 
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Assessment of Current Commercial 
Software Acquisition Approach 


• Strengths 

- Open Architecture Compatible with Usage of COTS Software 

- Much Less Formal Competitive Procurement Process 

» Includes Prior Performance as Major Selection Criterion 

- Trend Toward Joint Assumption of Risks Between Buyers and Suppliers 
Across Development, Operation, Maintenance and Modernization 

- Process is More Flexible 

- Shorter Cycle (Product Release) Times 

- Tailorable Level of Documentation and Oversight 

- Emphasis on Reuse and Tailoring Requirements to Existing Products 

- Beta Testing Widely Used 

• Weaknesses 

- Less Responsive to Continual Changes in Requirements 

- Less Assurance that Software Will Function Properly Under All Situations 

- Potentially Locked into One Vendor's Proprietary Application 

J 


The strengths and weaknesses of the current commercial approach to software development, 
acquisition and operation are summarized above. 
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Principal Reasons DoD Software 
Programs Get Into Trouble 



• Poor Requirements Definition 

- Lack of User Involvement in Development Process 

- Inability of Users to Foresee Benefits of Automation Without Incremental Capability 

• Inadequate Software Process Management and Control by Contractor 

• Lack of Integrated Product Teams 

- Failure to Establish "Team" With Vendors and Users 

- Little Participation of Functional Area Experts 

• Ineffective Subcontractor Management 

• Lack of Consistent Attention to Software Process 

• Too Little Attention to Software Architecture 

• Poorly Defined and Inadequately Controlled Interfaces Between Computer 
Hardware, Communications and Software 

• Assumption That Software Upgrades Can "Fix" Hardware Deficiencies 
(Without Assessment of Cost and Schedule Risks) 


• Focus on Innovation Rather than Cost and Risk 



Limited or No Tailoring of Military Specifications Based on Continuing 
Cost-Benefit Evaluations 


Over the last two decades, a number of DoD software development efforts have gotten into trouble, 
particularly in terms of actual costs and schedule vs. expected or predicted costs and schedule. The Task 
Force heard briefings from a number of DoD program managers where this was the case. Based on the 
specific programs discussed and on other inputs, the Task Force prepared a listing of the principal reasons 
DoD software programs have gotten into trouble as shown above. Certain of these reasons are a reflection 
of DoD practice. Others are tied to the software engineering level available at the time such programs were 
initiated. The Task Force believes that today’s software technology and practices can directly address 
many of the root causes for such past problems. 
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Resource Allocation: 
Where Does the Effort Go? 



Military 

Commercial 

• Engineering 

30% 

50% 

• Evaluation 

20% 

20% 

• Management 

15% 

10% 

• Meeting Support 

15% 

5% 

• Documentation 

15% 

7% 

• Customer/User Support 

5% 

8% 


Source: Magttavox and Capers Jones 


There are some indications that commercial development efforts have achieved better predictability 
and lower costs than DoD counterparts. The Task Force solicited quantitative data on this issue from both 
government and commercial software experts. The figure above summarizes one type of indicator of the 
difference between commercial and government projects (in terms of the percentage of the effort expended 
for different aspects of a typical development). 


18 







Comparison of 

Commercial and Government Projects 


Application Type 

Number of Projects 

Average Size (SLOC- 
New and Modified) 

Schedule/Time 

(Months) 

him 

Commercial 


26,000 

12.52 


Government 

75 

26,000 

15.1 


Percent Difference 



+ 17.09% 

+ 44.24% 

SmallEn^emugSystems 

Commercial 

V' N ' 

. IIS s s 

26,000 

17.4 

65.7 

259 

Government 

29 

26,000 

20.9 

117.3 

Percent Difference 



+ 16.75% 

+ 43.99% 

targe fnjvmatim Systems v 

H ■■■ 1111 


22 

150.2 

Commercial 

295 


Government 

21 

212,000 

25.6 

242.9 

Percent Difference 



+ 14.06% 

+ 38.16% 

l^jtopneeriugSystem 

Commercial 

* 56™ 

442,000 

45 

1736 

Government* 

7 

442,000 

52 

2740 

Percent Difference 



+ 13.46% 

+ 36.64% 

Summary of 

Average Percent Difference 



+ 15.34% 

+ 40.76% 


* Small sample may be statistically invalid. 
Source: QSM, Inc. 


This figures summarizes other quantitative indicators of the difference between commercial and 
government projects (in terms of the typical size of the code, time to develop the application and overall 
level of effort). 

It should be noted that the Task Force was unable to find reliable, quantitative data supporting the 
notion that commercial practices are more cost-effective than DoD practices. This lack of reliable 
indicators was a major concern of the Task Force. 
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3.0 MAJOR FINDINGS AND RECOMMENDATIONS 

3.1 PROCESS CREDIBILITY 



Process Credibility 



Findings 

• Attempts to achieve absolute fairness in competition in contracting 
(fair and reasonable pricing) have led to a lack of trust between 
government and individuals/contractors 

- Current DoD practice: 

» Is not compatible with commercial business practices 
» Is focused on contractor management/audit activities and costs 
» Hinders flexibility (managers take no personal risk) 

» Is not well suited to procuring complex, knowledge-intense, "first of a kind" systems 
» Is too costly 

» Does not prevent malfeasance or incompetence 
» Leads to adversarial relationships 

» Reduces reuse and contractor incentive to invest because of stringent data rights 
interpretation 

- Contractors must make profit on single contract; no long term relationship (DoD 
business practices do not view profit as a legitimate cost of doing business) 

- No individual fully understands or owns total process 

J 


The Task Force spent considerable effort on how the requirement for DoD to ensure public trust in 
its acquisition process influences DoD’s ability to employ “commercial best practices.” The Task Force 
found that attempts to achieve absolute fairness in competition in contracting have, in fact, led to a lack of 
trust between the government and individuals/contractors. DoD acquisition processes are focused on 
contractor audit activities to an extent that hinders the flexibility of Program Managers and contractors. 
DoD PMs are not incentivized to assume any personal risk associated with allowing for flexibility 
comparable to commercial practice. Criminal sanctions are a significant disincentive for such efforts. 

DoD’s acquisition system still does not prevent malfeasance or incompetence and leads to 
adversarial relationships rather than partnerships which are the norm within commercial industry software 
developments. One problem highlighted by the Task Force was that the current DoD system does not 
allow one individual or manager to control the total process, even for a specific project. This lack of 
control leads to a diffusion of accountability and hinders DoD’s ability to oversee complex “first of a 
kind” software developments. 
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Process Credibility 




Findings 

• Requirements and Source Selection Inflexibility 

- Vendors attempt to meet every requirement 

- Ease of protest 

- Requirements focus vendor on particular solution 

- Requirements have become alternative to professional judgment 

- Departures from requirements have caused protests and bad publicity 

• Price/Schedule/Functionality 

- In commercial sector, managers constrain 2 out of 3 (e.g., cost and schedule) 

- In DoD, managers constrain 3 out of 3 

• Constrained Communication During Solicitation 

- Questions are provided to competitors (can give away proprietary concepts) 

- Questions are often misinterpreted and answered incorrectly 

- Guarded way of asking questions limits substantive feedback 







Complicated Regulations 

- Restrict variety of proposals 

- Restrict competition and limit government options 

- Consume time and resources 

- Drive government employees to follow conservative procedures as safest path (inhibits "best value") 


Government-Unique Accounting Procedures and Audits 

- Audit requirements limit available vendors 

- Add substantial cost 

- Create need for separate accounting systems 

- Drive vendor to lowest labor cost solution rather than best value solution 

- Lead to rejection of good solutions based on value pricing vs cost-based pricing 



The viewgraph above lists important hindrances that the Task Force sees with regard to the 
adoption of commercial software practices. Many of the Task Force recommendations address these 
hindrances. 
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Process Credibility 



Recommendations 

• Make necessary changes to acquisition regulations 

- Have Program Managers Manage 3 of 3 (Price/Schedule/Functionality) But Only 
Constrain 2 of 3 

- Define Successful Performance on Contracts as Delivering Solution (with Predictable 
Price, Schedule and Functionality) Not Adherence to Government Processes, 
Procedures and Specifications 

- Do Not Require C-Level Specifications for Software Projects Developed in Ada 

- Prior to RFP, Government Should Perform Independent Market Analyses of Off-the- 
Shelf and Contractor Products to Assure "Best Value" Solution 

- Establish Mechanisms to Allow Both Current Ability to Perform as Well as Past 
Performance as Key Factors in Source Selection 

» Require Source Selection Evaluation of Development Contractors Through a Formal Software 
Process Capability Evaluation 

- Encourage Offerors to Demonstrate as Much Functionality as Possible as Part of Bid 
Without Eliminating Domain Knowledgeable Competition 

» Executable Architecture as a Minimum 
» Weight Heavily in Selection 




The Task Force makes the above recommendations with regard to process credibility. 


3.2 DOD PROGRAM MANAGEMENT 



DoD Program Management 



Findings 

• Program management does not encourage "80% solution for 20% cost" 

• Users often not significantly involved in process 

• Little emphasis on life-cycle issues (including maintenance and support) 

• Existing policies, methodologies, and procedures, and their implementation are 
inadequate 

- Little evidence that policies are influenced by actual experience and vice versa 

- Little effort to measure effectiveness and costs of policy directives 

• Some indication that DoD is migrating toward development and use of 
standards-based architectures 

• Program Managers lack incentives to allow tailoring of procedures to fit 
individual program needs or to develop "corporate" solution (e.g., employ 
common architecture or common software components) 


The Task Force found that the DoD program management system itself discourages the use of 
commercial best practices. These findings are not unique to software, but are particularly important for 
software given the significant cost savings associated with software reuse. There are few incentives for 
Program Managers (PMs) to develop or even employ corporate solutions (common architectures/software 
components), particularly if they are more expensive to acquire. Rather, each PM will tend to optimize 
his/her development for its own purpose. Further, it is difficult for users to be significantly involved until 
late in a software development process, unless some sort of prototype can be constructed. DoD PMs place 
little emphasis on life-cycle issues (such as software maintenance and support). Existing DoD-wide 
software policies, methodologies, and procedures, and their implementation by PMs are inadequate. 
There is little evidence that policies are influenced by actual experience and vice versa and there is little 
effort to measure the effectiveness and costs of policy directives. 
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DoD Program Management ( cont .) 



Recommendations 

• Establish Overarching Software Life Cycle Guidelines 



- Tools/Methods 

» Define Software Architectures to Enable Rapid Changes and Reuse 

» To Achieve the Benefits of Using Standards-Based Architectures, DoD Must Manage Programs Using: 

• Early system engineering 

• Iterative development 

• Proactive participation in development of these standards 

» Promote Development/Use of Community-Wide Metrics and Models (e.g., SEI's Capability Maturity Model) 


- Acquisition 

» Revise the Milestones for Software-Intensive Development 

• Address the need for a software-first philosophy 

• Provide for a layered software/hardware standards based architecture 

• Acquisitioin and life cycle planning should now separate hardware and software fieldings based on the business sense of the specific project 

» Require Early Interaction Between User, Acquisition Agent, and Developer; Identify and Get Early User Involvement 
» Apply Evolutionary Development with Rapid Deployment of Initial Functional Capability 
» Encourage Competition of Technical Approach vs. Cost 

» Provide Incentives and Guidelines to Encourage Software Reuse (Architecture-Based Reuse) 

» Reduce Documentation and Review Requirements for "Mature" Companies (i.e.. Companies Determined to Be "Mature" 
Through Evaluation Mechanisms) 

» Tailor operational testing to develop DoD "Beta Test" philosophy 

• Allow fielding of software direct from test beds with user consent 


Have Program Manager Stay with Programs at Least Through Beta Testing to Maintain Continuity of Understanding of 
Original Nuances in Requirements 


The Task Force makes the recommendation that DoD establish overarching software lifecycle 
guidelines directed at facilitating program manager employment of commercial practices and software and 
that a DoD-wide effort be made to oversee implementation of these guidelines. The tools, methods and 
acquisition approaches recommended by the Task Force are listed above. 
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3.3 DOD PERSONNEL 



DoD Personnel 



Findings 

• A shortage of sufficiently qualified software personnel 
currently exists at all levels within the DoD 

- Expertise for software acquisitions, software evaluations, and software 
maintenance/support 

- Expertise to represent DoD (customer) interests with commercial sector 

- Expertise in domain software design and applications 

- Expertise in software technology to develop policies, standards, and 
guidelines 

- Expertise in software program management 


v y 

Based on the inputs provided, the Task Force found that a shortage of sufficiently qualified 
software personnel exists at all levels within the DoD. Without personnel who are highly qualified in 
modem software practices, DoD will not be as capable of effectively exploiting complex software within 
its systems. This personnel shortage has been a major contributor to the problems that have arisen in past 
DoD software development programs. 
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DoD Personnel (Cont.) 


Recommendations 

• Establish DoD-wide software program management education and training 
initiative 





- Change DSMC and IRMC courses for PMs to reflect best commercial practices and other 
recommendations of this Task Force and Provide for changes to reflect the dynamics of the 
software industry 

- Develop and provide interactive training tools for senior managers to perfect software 
management skills 

- Rotate government and contractor personnel between PM and developer organization to build 
understanding and trust; encourage use of IPA's from industry 

- Incorporate software management principles in senior management education and seminars 
(including senior services colleges) 

- Provide mechanisms for keeping software expertise current in the workplace 

Develop Acquisition Managers with software program management expertise 

- Integrate software-qualified personnel into senior acquisition staff 

Establish Norms for the Number of Software Experts in Program Offices 

Upgrade Educational Requirement for Personnel Assigned to Acquisition, 

Management, Development and Oversight of Software Intensive Programs 


Develop Expertise in Analysis of Domain Software Design 

- Promote Software Reuse in the Design 



The Task Force makes the above recommendations with regard to DoD software expertise of its 
personnel. The Task Force strongly recommends an emphasis on increasing the capability of its personnel 
in modem software practices and techniques. 


X 



3.4 USE AND INTEGRATION OF COMMERCIAL OFF-THE-SHELF SOFTWARE 


Use and Integration of 
Commercial Off-the-Shelf Software 


Findings: 

• DoD Does Not Normally Perform COTS Market Analyses Incident to 
Requirements Definition 

• DoD is Beginning to Exploit Evolutionary Development Approaches 

• Tools/Methodologies Are Evolving to Facilitate Use of COTS 

- Computer Aided Software Engineering (CASE) 

- Object oriented methods 

- Reuse repositories 

- Integrated product teams 

- Integrated risk management 


v s 


DoD does not routinely perform COTS software market analyses during the requirements 
definition phase of an acquisition program. Nor does it employ prototypes or simulations of new 
capability in a way that could influence the requirement process. Rather, requirements are evolved in a 
manner that is disconnected from the availability of commercial applications and are not typically 
influenced by such available capability. This situation is true for both hardware and software. Today’s 
technology can facilitate early interaction between users and available capability, particularly in software. 
Further, DoD is beginning to exploit evolutionary software development approaches that could provide for 
the inclusion of existing commercial software functionality into early prototypes and allow users to test 
such capabilities. Modem software tools and methodologies have evolved that facilitate use of COTS, 
such as those listed above. 
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Use and Integration of 
Commercial Off-the-Shelf Software 



Findings (Cont.) 





DoD has not fully identified the pros and cons associated with the use of COTS 

- Pros 

» Saves money in design and development of those components 
» Can Significantly Reduce Development Risk 

• Support is Concern 

» Selection of COTS products early in project cycle will enable requirements to be driven by commercial software 
capabilities 

» Very useful in rapid prototyping 

- Cons 

» Commercial products must still be integrated into system and qualified with system 
» Difficulty in Configurement Management and Support for Older Releases 
» Additional testing may be required to qualify system with commercial components 
» Subsequent releases determined by vendor, not DoD 
» Security Aspects of Use of COTS Not Well Understood 

• DoD not addressing multiple aspects 

- Development environment 

- Tactical computer program 

- Virus protection 

- Commercial Espionage 

• Classified software systems a problem for commercial companies 

In addition, DoD has not determined when to use COTS 


- Commercial Off-the-Shelf Software Products May Not Apply to All DoD Systems 

- Should be used as is - avoid tailoring or special features 

- Most weapon system/real-time application software for DoD will not exclusively be ''custom,” but 
will involve some COTS 



In general, DoD has not identified the pros and cons associated with the use of COTS. DoD must 
learn how to balance the cost savings associated with design and development of commercial software 
products and the significantly reduced development risk with the concern for longer term support and 
system security. Commercial products must still be integrated into and qualified within each defense 
system and there is difficulty in configuration management and support for older releases. The latter point 
is important since many DoD software systems are not deployed before a commercial software product is 
retired or replaced. DoD is not addressing many aspects of the use of COTS software: 

- Development environments 

- Virus protection 

- Commercial espionage 

In essence, DoD has not developed a corporate way to decide when to use COTS software. 
Commercial off-the-shelf software products do not apply to all DoD systems. Most weapon system/real- 
time application software for DoD will not exclusively be "off-the-shelf." Integration and configuration 
control for COTS software then become important concerns. Further, DoD must learn to use COTS so 
that it avoids tailoring or special features. 
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Use and Integration of 
Commercial Off-the-Shelf Software 


Recommendations 




. • 



Require Trade Studies and Analysis of the Use of COTS in DoD's Software Acquisition 
Process Where Effective 

- Performed by Acquiring Organization as Essential Part of Defining Requirements and in Rapid Prototyping 
Situations 

» Employ Broad Agency Announcements or Similar Contractual Approaches to Facilitate Such Studies 

- Use of COTS Appropriate When: 

» Defining Requirements 
» Rapid Prototyping Situations 

» Not Required to Tailor COTS Source Code to Application 
» Not Required to be Error-Free 

» COTS Software is "Close Enough" to Tailor Requirements 

Establish "Customer Friendly" Application-Specific Information Technology 
"Component Stores" 

- Generic Architectures for Specific Domains 

- Rapid Requirements Definition Process and Prototyping 

- Reusable, Prequalified Components 

- Assemble Systems Rather than Develop Them 
Reduce Lead Time 
Security is not Paramount 

Increase tech base funding for security audit tools for systems employing COTS 

Capitalize on Innovative Cost-Effective Techniques for Acquiring and Using COTS 
Software Products 

- Such as Use of Enterprise Licenses 



Given these findings, the Task Force makes the above recommendations with regard to DoD use 
and integration of commercial off-the-shelf software. The Task Force sees great benefit to be gained 
through exploitation of COTS software; however, DoD must develop corporate approaches to the use and 
integration of COTS software, if it is to gain this benefit. 
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3.5 ACQUISITION 


Acquisition 



• Findings 

- Acquisition practices have led to: 

» Distance between user and developer 
» Limited participation by commercial software companies 

- Adherence to DoD regulations for reviews and documentation is increasing 
software costs 

» DoD software costs are estimated to be increased by at least 20 % over 
commercial best practice 

» Commercial best practice requires much less documentation than DoD 

- Funding for maintenance planning/execution starts late 

» Maintenance assumed organic; inhibits teaming/partnerships 

- Acquisition focus is on mandatory "how to" specifications and standards 
rather than the product (what) 

» Lengthens process and adds costs 
» Discourages harmonization with commercial practice 
» Creates adversarial relationship 

- Acquisition process does not reward development of reusable software 


The Task Force was concerned that the specific acquisition and contracting approaches used by 
DoD inhibit use of commercial practices and software. The Task Force findings in this area are shown 
above. Strict adherence to DoD regulations for reviews and documentation is increasing software costs. 
DoD software costs are estimated to be increased by at least 20% over commercial best practices. The 
DoD focus on detailed technical specifications has lengthened the process, added costs, discouraged 
harmonization with commercial practice, and created a highly adversarial relationship between the 
Government and industry. There is a strong belief that certain commercial companies or divisions of 
companies have opted to stay away from government contracts due to the complexity of the acquisition 
rules and regulations. 




3D 


Recommendations-Acquisition 




• Recommendations 



- Implement the following acquisition approach: 

» Establish acquisition focus on functionality and consistency with 
"commercial best practice" 

» Revise procedures encouraging interaction between user and developer 
and achieving early functionality 

» Minimize DoD regulations for review and documentation that are 
different than "commercial best practice" 

» Require planning for maintenance at beginning of development process 

» Provide government funded vehicle in contracts to incentivize 
development of reusable SW 

- Review all existing military standards and military specifications 
pertaining to software development and documentation, for 
continued applicability, such as DoD-STD 2167 



To this end, the Task Force makes the above recommendations with regard to DoD software 
acquisition practices. In essence, these recommendations can move DoD toward an acquisition approach 
that is more consistent with commercial approaches, while not requiring changes in statutes. 
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Proposed Software Life Cycle 




Pre-Prososal Demos & 
Market Analysis 



Proposal & Demo 
Simulation Capability 


Version 1.0 
Basic Functions 


Version 2.0 
Improved Functions 


Version 3.0 
Improved Functions 




To implement this recommendation, the Task Force proposed that DoD evolve a more appropriate 
software life cycle approach as depicted above. This approach provides for early capability (even in the 
initial bidding process) and for a gradual enhancement in capability over time. 
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Acquisition Approach 




Developing the RFP 




Contract Proposal 

RFP Response 

Qualified Software Organization 


Integrated Top Level Specification 

• Performance 

• Environment ; 

• Test^Validaiion 


Domain Experience 
Skills Matrix 

Development Environment 
Process Control 
Peer Reviews 

Metrics and Milestone Plan 
Reuse Plan 

Program Managment Plan 


, . 


Proposal Demonstration 


Selection 

• Acquiring Agency 

— Evaluate Technical Solution and Management Plan 
— Evaluate SW Process Maturity and Past Performance 

• Users 

— Define Requirements 
— Evaluate Demonstrations 
— Heavy Participation in Selection 



The Task Force proposed approach to competition is shown in the above figure. The core 
government role in such an approach would be to: 

• Develop the RFP (no “How To” statement of work; no management CDRLs; no government in- 
process approvals) 

• Provide an integrated “Top Level” specification (architecture, COTS/reuse, software engineering 
environment and test/validation approach). 

• Employ software metrics as a key determinant, 

• Evaluate proposed technical solutions and the proposed management plan 
The contractor would then: 

• Provide an execution plan, management controls and progress milestones/metrics 

• Describe an in-place, mature software development organization and relevant domain experience 

• Provide a skills matrix describing personnel to be employed 

• Identify a robust development environment and describe applicable prior experience 

• Describe automated process control software 

• Describe the extent to which peer inspections will be used 

• Provide a metrics usage plan and purposes for which they will be used 

• Provide specific reuse and program management plans 

• Propose specific architecture(s) in executable code 
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3.6 SOFTWARE ARCHITECTURE 



Software System Architecture: 
The Missing Link? 



What is software architecture? 

• Software architecture consists of: 

- Software system components 

- The relationships among those components 

- Rules for their composition (constraints) 

• Defining document would address: 

- System functionality 

- Software system components 

- Interfaces, standards, protocols 

- Execution model 

- Data flows 

- Control flow 

- Critical timing/throughput aspects 

- Error handling 


Software architecture was emphasized by the Task Force as a means for achieving important ends. 
Software architecture consists of software system components, the relationships among these components 
and the rules for their composition (constraints). To use architecture as a management tool, DoD needs to 
define: system functionality along with software system components to be employed; interfaces, 
standards, and protocols to be employed; and the execution model to include data flows, control flow, 
critical timing/throughput aspects, and error handling approach. 
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Software System Architecture (Cont.) 


"\ 


Findings 

• Why is it important? 

- Essential for effective management over the lifecycle 

» Software system lifecycle costs ~ 65% for maintenance 
» - 65% of maintenance costs are due to changes/modifications/upgrades 

- Software architecture is a prime enabler of flexibility and reuse 

- Well-formulated architecture might reduce costs of changes/upgrades by 30- 
50% ($4-$7B/year assuming software expense ~ $30B/year) 

• Why doesn't it play a larger role? 

- Focus is usually on initial cost schedule, functionality - not lifecycle 

- 2167A reinforces this approach - requires proof that design satisfies 
functionality 

- Difficult to specify, test, etc. 

- Not well understood 




Software architecture is a prime enabler of flexibility and reuse, and a well-formulated architecture 
might reduce costs of changes/upgrades by 30-50% ($4-$7B/year assuming software expense 
~$30B/year). Software architecture has not been emphasized because PM focus has usually been on initial 
cost, schedule, and functionality and not on the life cycle. DoD-STD-2167A reinforces this approach by 
requiring only proof that a design satisfies the required functionality. 
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Software System Architecture (Cont.) 



Findings (Cont.) 

• Little emphasis on architecture in DoD software programs or regulations 

• Impact of software architecture issue 

- DoD not benefiting from architecture as a key tool for: 

» Evolutionary development 

» Early (and often) involvement of users with functional capability 
» Ability to include changing commercial technology 
» Reuse 

» Facilitating requirements change with minimum cost and schedule 
» Facilitating product line management 

• Insufficient Progress in: 

- Developing models/standards for domain specific software architectures 

- Open system architectures work 
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There is currently little emphasis on architecture in DoD directives or regulations. As a result, DoD 
is not benefiting from architecture as a management tool. Further, the Task Force sees insufficient 
progress in developing models and standards for domain specific software architectures. 
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Software System Architecture (Cont.) 


Recommendations 

• Emphasize Use of Software Architecture 

- Establish model and context for architecture selection 

» Standards-based with emphasis on "implementable" 

» Require vendors to propose, manage and control the architecture 

- Require delivery of software architecture definition as first step in any 
software acquisition 

- Foster migration strategies at architecture level 

• Assign responsibility within Government for domain analysis and product 
line developments 

• Provide expertise and resources to ensure coordinated DoD participation 
in commercial/intemational standards bodies and users groups 




The Task Force makes the above recommendations in order to facilitate greater use of software 
architecture as a management tool for DoD software programs and activities. In particular, the Task Force 
sees great value in requiring the delivery of software architecture definition as a first step in any software 
acquisition. Where possible, such software architecture definition should be operational (i.e., executable). 
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Software Technology Base (Cont.) 




Recommendations 

• Provide for the evolution of the DoD Software Technology Strategy to align 
with emerging commercial technology and practices 

• Emphasize Technology Transfer (External and Internal) 

- Fund technology transfer programs in such topics as: 

» Architectural principles, 

» Architecture description languages, 

» Standard interfaces, and 
» Integration technologies 

- Initiate demonstration programs (e.g., ATDs) to facilitate software technology 
insertion into systems. Examples of candidate criteria: 

» Open standards 
» Use of COTS and GOTS 

» Frequent releases to include a number of users 
» Multiple platforms 

» Satisfies commercial standards and interoperability standards across DoD 
organizations 

• Initiate formal data collection and analysis 


The Task Force makes the above recommendations with regard to DoD software technology base 
investments. The Task Force supports a DoD technology base program that is more closely aligned with 
the wide range of similar efforts ongoing in commercial organizations. 
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4.0 


SUMMARY 


4.1 “ATTA PERSONS” 

r 


"Atta Persons " 



• Electronic Systems Command Software by Assembly of Modules 

- Software "Component Store" 

• A-109 Acquisition Methodology 

• Use of Commercial Practices and Products in Reserve 
Component Automation System (RCAS ) 

• Software Reuse, Prototypes and User Involvement of Global 
Command and Control System (GCCS) Initiative 

• Perry-Paige Initiative Toward Migration Systems 


V J 

The Task Force identified several ongoing efforts worthy of note. 

• The Air Force Electronic Systems Command (ESC) has defined an approach for software system 
development through the assembly of pre-qualified software modules (PRISM). ESC is pursuing 
the development of such software modules. The Task Force was very supportive of this program. 

• Office of Management and Budget (OMB) Circular A-109 outlines an approach for major Federal 
system acquisitions that encourages: definition of top level needs vs. detailed specifications; 
exploitation of innovative private sector contributions and use of early competitive demonstrations 
of competing approaches. These all are acquisition attributes recommended by the Task Force. 

• The Reserve Component Automation System (RCAS) is an ongoing MIS development to support 
the reserves. The program has successfully employed the A-109 acquisition approach and 
extensively used commercial acquisition practices and products. The Task Force commends this 
approach. 

• The Global Command and Control System (GCCS) is an initiative of the Joint Staff (J3 and J6) to 
provide vertical and horizontal interoperability of combat information systems across Services, 
Combatant Commands and Agencies. It was highlighted for its software reuse approach, its use of 
prototypes and its emphasis on operational user involvement. 

• The Perry-Paige migration systems initiative has established a focus on selecting a set of target 
computing systems (including MIS, C3I and embedded) towards which DoD will aim. This 
migration strategy will enable a more cost-effective DoD investment in software across the life- 
cycle. 
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4.2 OVERARCHING RECOMMENDATION 



Overarching Recommendation 



• SECDEF Assign USD (A&T) the Responsibility for DoD-Wide Software 
Technology, Policy, Practices and Acquisition 


• This Responsibility Includes Allocating Resources and Assigning 
Responsibility for: 

- Drafting and Institutionalizing a New Software Acquisition Policy That 
Includes the Recommendations of this DSB Task Force 

- Creating Incentives and Ensuring Compliance to the Policy 

- Creating Terms and Conditions and Financial Rewards to Maximize the 
Ability of DoD to Realize the Benefits of Commercial Best Practices 

- Requiring Source Selection Evaluation of Development Contractors Through 
a Formal Software Process Capability Evaluation 

— Advising the Acquisition Executive on Matters Concerning Software 
Technology, Acquisition and Architecture for Major Programs 


V 


- Maintaining a Digest of Lessons Learned and Best Practices and 
Communicating the Same to Program Managers and Contractors 



Throughout its deliberations, the Task Force acknowledged that the issues associated with defense 
software were applicable across the spectrum of DoD software intensive systems. The Task Force also 
frequently learned of obstacles based, in part, on the current dichotomous DoD structure associated with 
software technology, policy, practices and acquisition. In order to ensure that the DoD reap maximum 
benefit from its recommendations, the Task Force formulated the above overarching recommendation. 
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Overarching Recommendation (Cont.) 




• In Carrying Out This Responsibility, Consider Forming an Executive 
Council 

- Including the DDR&E, the ASD (C3I) and Appropriate Representatives from 
the Services and Defense Agencies 

- Provide Supporting "Process Action Team" to Assist in Implementation 




The Task Force also identified the mechanism by which its overarching recommendation could be 
readily implemented. 





APPENDIX A 
TERMS OF REFERENCE 



THE UNDER SECRETARY OF DEFENSE 
WASHINGTON, DC 20301-3000 



ACQUISITION 


DEC 0 6 ' 1993 


MEMORANDUM FOR CHAIRMAN, DEFENSE SCIENCE BOARD 

SUBJECT: Terms of Reference — Defense Science Board Task Force on 

Acquiring Defense Software Commercially 


You are requested to form a Defense Science Board Task Force 
on Acquiring Defense Software Commercially. Determine the 
conditions under which the procurement of defense software (i.e., 
operational software, support software, and software tools) can 
appropriately use commercial practices and develop a strategy for 
defense software procurement that substantially incorporates such 
practices. Specifically address within this strategy DoD use of 
commercial software products. 

The scope of this effort should include all DoD systems that 
are software intensive. It should address all stages in the life 
cycle of a software component from initial procurement to 
evolutionary upgrade of software or of software/hardware 
combinations. This commercially-based strategy should not be 
constrained by existing DoD standards; it should be viewed as a 
coexisting alternative to, rather than replacement of, the 
current DoD procurement strategy. Accordingly, you should 
compare your proposed strategy to the current DoD strategy to 
indicate circumstances in which e^ch strategy is most beneficial. 

To assess the appropriateness of DoD use of commercial 
practices, the Task Force should identify and apply objective 
measures such as elapsed development time, software life-cycle 
cost, management risk, as well as measures of software product 
quality. 

The Task Force should consider at least the following 
topics: 


• Technical: state-of-the-art and best commercial 

practices, development tools, reusable software components, 
and techniques and tools for tailoring commercial software 
components for use in defense systems. 

• Management: DoD management of the commercial development 

process, software risk management techniques and supporting 
tools, minimum delivery time, affordability, maintenance 
after product delivery, post-deployment product enhancement, 
software process support tools, quality and assured 
availability, and use of development and maintenance tools. 


• Contracting: technical data rights, intellectual 

property rights, liability, alternative forms of procurement 
agreements, and incentives for creation of reusable software 
components as well as their subsequent reuse. 

The Director, Defense Research and Engineering will sponsor 
this study. Dr. George H. Heilmeier and Dr. Larry E. Druffel 
will serve as Co-Chairmen. The Office of the Director, Defense 
Research and Engineering will provide the necessary funding and 
support contractor arrangements. The Executive Secretary will be 
Ms. Virginia L. Castor. Commander Robert C. Hardee will be the 
Defense Science Board Secretariat representative. It is not 
anticipated that this Task Force will need to go into any 
"particular matters" within the meaning of Section 208 of Title 
18, U.S. Code, nor will it cause any member to be placed in the 
position of acting as a procurement official. 




APPENDIX B 
BRIEFINGS 

PROVIDED TO THE TASK FORCE 


Briefings Provided to the Task Force 


Joint Staff Views on How DoD Can Adopt Commercial Software 
Practices 

MG David Kelley, Vice Director J6 

Army Views on How DoD Can Adopt Commercial Software Practice 

LTG Peter Kind, Director, Information Systems for C 4 

Naval Views on How DoD Can Adopt Commercial Software Practice 

RADM John Hekman, Commander, NISMC 

Air Force Views on How DoD Can Adopt Commercial Software 
Practices 

Mr. Lloyd Mosemann, Deputy Assistant Secretary of Air 
Force (Communication, Computers & Support Systems) 

DISA Views on How DoD Can Adopt Commercial Software Practices 

Dr. Mark Scher, Director of Infrastructures, Defense 
Information Systems Agency 

DoD 5000 Series Regulations 

Mr. Gene Porter, Director, Acquisition Program Integration 

DoD 8000 Series Regulations 

Mr. Harry Pontius, Director, C^I Policy 

DoD-STD-2 1 67 A/MIL-STD-498 

Dr. Singh, Senior Manager, Space & Naval Warfare Systems 
Command 

Draft White Paper on Software Acquisition 

Dr. Jack Ferguson, Program Manager, Software Engineering 
Institute 

Data Modeling vs. Data Standards 

LTG Peter Kind, Director, Information Systems for C 4 

Case Study: B2 

Mr. Fred Schwartz, Director of Engineering, 

Case Study: Ensemble of Real-time Software Systems 

MajGen Israel, Director, Defense Airborne Reconnaissance 
Office 

Case Study: Reserve Component Automation System (RCAS) 

MG Gary Stemley, Program Manager for RCAS 

Case Study: AEGIS 

CAPT. Richard Cassidy, AEGIS Technical Director 

Case Study: PRISM 

Mr. Robert Kent, ESC/ENS 

Case Study: BMC3 

Lt Col Robert Phelps, BMDO 

Boeing Views on DoD vs. Commercial Software Practices 

Mr. John Hanson, Director, Systems Software & Engineering, 
Boeing ! 

McDonnell Douglas Views on DoD vs. Commercial Software Practices 

Mr. L. George Hite, Senior Manager, McDonnell Douglas 

IBM Views on DoD vs. Commercial Software Practices 

Mr. Dave Coyer, Program Manager, IBM Federal Systems 
Company 

Object Technology and a COTS Product Called "SNAP" 

Mr. Joe Fox, Chairman, Template Software Inc. 

Procurement Regulations/Issues 

Mrs. Eleanor Spector, Director, Defense Procurement 
OUSD(A&T) 

Integrated Computer Aided Software Engineering (ICASE) Program 

Col Gary Case, ICASE System Program Director 

Computer Sciences Corporation Case Study on Commercial Software 
Development 

Mr. Daniel Kemp, Senior Partner, Computer Sciences Corp. 

Commercial Software Acquisition Practices 

Mr. Alex Morrow, General Manager, Lotus Development 
Corp. 

Experiences in DoD Software Maintenance 

Mr. Bobby McDonald, Deputy Director, Electronic Warfare 
Directorate, Warner Robins ALC 

MITRE Corporation Experiences in Exploiting Commercial Practices 

Mr. Steven Crisp, Dept. Head, MITRE Corp. 

FA A Study on the Use of Commercial Software 

Mr. John Stenbit, Vice President & General Manager, TRW 
Systems Integration Group 

NSIA Study on the Use of COTS Software 

Gen. William Richardson, USA(Ret), Executive Vice President 
Army Programs, Burdeshaw Associates, Ltd. and Ms. Linda 
Connor, Program Manager, Rockwell International 

DoD Software Reuse Initiative 

Ms. Linda Brown, ODASD(IM)/Information Technology 

MITRE Study on Feasibility of Using a Software Acquisition Maturity 
Model in DoD 

Ms. Judith Clapp, Division Assitant, The MITRE Corp. 

Software Management Metrics and Reliability 

Mr. Douglas Putnam and Mr. Lawrence Putnam, Jr., 
Quantitative Software Management, Inc. 

Innovative Techniques in DoD SW Acquisition: F-22 Integrated 
Product Team Approach 

Colonel Robert Kayuha 

Legal Impediments to Acquiring Defense Software Commercially 

Mr. Robert Gorman, OSD General Counsel 
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APPENDIX C 

COMPARISON OF SOFTWARE 
ACQUISITION METHODS 


Comparison of Software Acquisition Methods 1 


Requiremen 

ts Definition 

Best Commercial Practice 

Current DoD Practice 

Requirements based on strategic plan and market 
analysis. 

Requirements based on using command Mission 
Need Statement, master plans, and top-level 
certification. 

Requirements based on life-cycle resource 
constraints. 

Requirements based largely on annual budget 
resource constraints. 

Detailed requirements generated by interdisciplinary 
team including users, domain experts, and system 
engineers. 

Detailed requirements generated by buyer in 
collaboration with user. Team generally includes 
domain experts and acquisition personnel. 

Buyer, user, and vendor are a team. Attitude of 
partnership, trust and cooperation. Presumption of 
trustworthiness for reputable commercial 
organizations. 

"Us vs. them” mentality about contractors. 
Government thinks in terms of control, 
accountability, detailed auditing, and double 
checking. Presumption that contractors cannot be 
trusted. 

Functional specification is modified by knowledge 
of availability of existing products. 

Functional anchor performance specif ication; little to 
no regard for existing products. 

Vendors involved early in study, analysis and 
prototyping with emphasis on reuse and evolution 
of existing systems. 

May contract for prototypes, but contractor 
involvement in pre-award discussions is 
discouraged. 

Need is based on business case and decisions are 
based on return on investment. 

("time to make”) 

Need based on Mission Need Statement; decisions 
based on need, economics, and politics, ("time to 
field”) 

Efficient decision processes. 

Decision processes formal and time-consuming. 

Level of documentation is negotiable based on 
individual user needs and complexity of system 
being developed. 

Extensive (often redundant or unnecessary) 
documentation required under 2167A. 

Tailoring of documentation requirements is often 
minimal or discouraged. 

More detailed analysis of cost versus feature. 
Dropping lower value/higher cost options or 
reducing requirements is practiced. 

Little or no requirements reductions on high cost 
items. 

More requirements trade-off decisions (involving 
complexity and schedule) for reduced time to field. 

Very little flexibility to trade-off requirements creep 
versus complexity and schedule. 

Selected vendors may assist in preparing 
specifications. 

Vendors not involved in preparing specifications. 

Tools used to create system models for use in 
requirements definition; e.g., GUI Building. 

Requirements definition based on Mission Need 
Statement. 

Flexibility allowed in choice of programming 
language. 


Evolutionary and incremental approach favored. 

Requirements defined up front with little flexibility 
for modifications. 

Summary 

Commercial is more flexible and open between users and suppliers, and requirements are based on a 
strategic plan. In the commercial world, there is more willingness to adjust requirements based on 
availability of products and thus to filed a system sooner and evolve it to include more capability at 
significant cost savings. 


1 This appendix was initially derived from a White Paper on software acquisition methods prepared by the Software 
Engineering Institute. The resulting content represents the consensus of this Task Force. 
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Vendor Selection 


Best Commercial Practice 

Current DoD Practice 

Solicit multiple (but not all) qualified vendors -a 
selected few. Encourage teaming with a view to 
attaining a long term relationship that covers the 
entire life cycle and fosters trade-offs in cost and 
schedule. 

Solicit all possible vendors. Vendor proposals must 
meet 100% of requirements. Teaming seldom 
encouraged; development and maintenance usually 
separate entities. 

Compare vendor history and experience. Maintain 
long-term relationships. 

Can compare previous performance, but normally 
can't have long-term relationships. 

The organization that will be responsible for a 
system over its full life cycle is heavily involved 
from the beginning. 

Maintenance organization not usually involved in 
vendor selection process. 

Use site visits and demonstrations to gain 
knowledge of vendor capabilities. 

Site visit only by capability evaluation team, or 
other expert teams. Visits are very structured. 

Negotiate for best values based on: (1) trade-offs of 
costs and requirements licensing; and (2) 
consideration of both vendor and buyer best 
interests. 

Negotiation based on lowest cost and shortest 
schedules, endangering maturity of finished 
product. 

Overall goals: (1) obtain product at reasonable cost 
as soon as possible; and (2) achieve the business case 
for the system. 

Overall goal: Obtain lowest cost product that 

rigorously meets all requirements, but be fair. 

Relatively few review and approval steps once 
vendor is selected. 

Review and approval process more structured and 
complex once vendor selected. 

Past performance weighted heavily (sometimes 
primary factor) in selection process. 

Past performance considered, but only as a minor 
factor. 

More flexibility in vendor selection based on metrics 
and overall assessment. 

Selection of vendor forced by use of pre-defined 
metrics for proposal evaluation. 

Modifications made as procurement proceeds in 
order to get best results. 

Change difficult once process begins. 

Summary 

Very different processes with commercial much more flexible, but with no requirement for fairness, or to 
maintain the public trust. Commercial encourages vendors to offer best solution, but solution may not 
meet 100% of the requirements. Teaming and long-term relationships are more easily accommodated by 
industry. 
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Development Process 

Best Commercial Practice 

Current DoD Practice 

Vendor often tailors existing systems and uses 
COTS. System designed to fit in a defined product 
or product line architecture. 

Varies with application. Some systems use COTS. 
However, usually a new system that doesn't reuse 
legacy software. Unique systems are built with little 
regard for architecture. 

Buyer may have heavy involvement in design and 
development as part of the team (Integrated Product 
Development team). 

Formal, structured spiral, or waterfall model. Buyer 
oversees, but team approach is not usually 
emphasized. 

Reviews typically informal and stress progress 
against goals. 

Reviews usually very formal and include technical 
design details in addition to progress metrics. 

Buyer actively involved as co-participant in 
management of technical details. 

Micro management of technical details. 

Heavy user involvement. 

Limited user involvement. Heavy buyer 

involvement. 

Vendor embraces one or more industry standards 
which improves interface and integration with COTS 
products. 

Government and industry standards called out. Not 
all government standards enhanced by COTS 
products. 

Buyer requirements may be translated to more 
"general purpose” requirement for potential 
software reuse. 

Tailored system; little, if any, focus on designing in 
reusable code. 

Management reviews and degree of oversight are 
commensurate with size and risk of program. 

Notably more detailed reviews and oversight 
performed. 

Prototyping common, with joint applications 
development teams (user and developer) working to 
clarify requirements and incorporate new 
requirements that do not affect cost or schedule. 

Prototyping seldom used. 

Use of flexible architectures allows insertion and 
plug and play of COTS products. 

Use of MIL standard computers and legacy 
instruction set architectures restricts new 
development. 


Summary 

Commercial more flexible with likelihood of a team approach and is biases toward reuse and tailoring of 


existing systems. Multi-year acquisitions not re-justified each year. Product improvements are anticipated. 



C-3 









Business Practices 

Best Commercial Practice 

Current DoD Practice 

Informal contracting, joint ventures, partnerships 
with mutual economic benefit and vested interest in 
success. 

Difficult to write contract to motivate contractors to 
reduce cost; expensive to terminate contractors. 

Oversight built on established relationships. 

Burdensome cost accounting procedures required; 
extensive oversight, reporting, and documentation 
requirements. 

Can hire and fire vendors and managers. 

Government personnel regulations, policies, and 
practices determine qualifications of its managers, 
rotations of assignment, and training. 

Multi-year effort and funding. 

Multi-year effort. Yearly, unpredictable funding. 

Summary 

Commercial practice more flexible with greater incentives. 


Integration Testing and Delive 


Best Commercial Practice Current DoD Practice 


Unless system is for a. new plant, then there are Similar "cut-over” or transition issues, 
major "cut-over" issues. 


Sometimes difficult to assemble complete system in Usually integrate system in laboratory prior to 
laboratory environment due to cost. operational testing. 

Development testing vs. operational testing via 
statutory mandate. 


Beta testing widely used to quickly find errors. Little beta testin 


Ultimate acceptance authority rests with buyer, not a Structured, specified operational testing conducted 
separate organization. by separate authority. Acceptance authority rests 

with buyer. 


Buyer/user/vendor are a team. Emphasis on DoD as oversight and approval 

authority. 


Summary 

Integration and functional testing seem appropriate to the need. DoD use of separate test agency adds time 
and complexity. Absence of beta testing increases cost to DoD. 

















Maintenance 

Best Commercial Practice 

Current DoD Practice 

Organic support shifting to outsourcing or vendor. 

Organic support, with reluctance to be dependent on 
vendor. Use of depot maintenance makes 

interoperability issues more manageable. Also, 
must be responsive to user for critical systems. 

Level of discrepancy reporting required is based on 
user needs. Problem resolution usually delayed 
until next major release of software. Buyer has 
limited power to implement fast resolution of 
problems. 

Bureaucratic and paper-intensive discrepancy 
reporting and change control board often imposed. 
Award fees may be tied to problem workoff, 
resulting in fast resolution of problems. 

COTS solutions take advantage of economics of 
scale since maintenance costs are leveraged across 
multiple buyers. 

Unique software requires custom maintenance all 
borne by the single buyer. 

Summary 

The DoD and industry currently have different requirements, and trends are moving apart. However, DoD 
is currently reevaluating its practices for hardware systems and perhaps should also reevaluate for software. 
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APPENDIX D 


PROPOSED ACTION PLAN FROM 
SERVICE SOFTWARE EXECUTIVE 

OFFICIALS 



JOINT RECOMMENDED PROPOSED ACTIONS 

I. DESIGNATE SINGLE, COMMON DOD SOFTWARE MANAGEMENT OFFICIAL 
(ACTION OFFICE (USD)) 

A. ESTABLISH DEPUTY ASSISTANT SECRETARY OF DEFENSE 
(SOFTWARE) 

B. PROVIDE SUFFICIENT RESOURCES FOR MISSION 

II. SOFTWARE ACQUISITION AND LIFE-CYCLE MANAGEMENT (ACTION OFFICE 
NEW DASD) 

A. PROMOTE REUSE BASED ON DOMAIN MANAGEMENT 

1 . Define domains of interest or "areas of business" 

2. Assign responsibilities to manage the domains 

3. Establish and manage common architectures within the domains 

B. REDUCE MONITORING OF MATURE COMPANIES 

1 . Identify criteria for assessing/determine process maturity 

2. Consider maturity in source selection 

3. Allow sole source extension for high quality vendors 

C. PROMOTE GOVERNMENT/INDUSTRY TEAMING 

1 . Reduce Documentation Requirements to a minimum 

2. Provide for electronic delivery/evaluation/exchange 

D. MANAGE RISK 

1 . Mandate the use of market research 

2. Use metrics effectively for management 

3. Develop Risk Management Disciplines 

4. Stick with a winner/punish poor performers 

III. POLICY AND STANDARDS 

A. ADOPT COMMERCIAL STANDARDS (ACTION OFFICE(ASD(C3I))) 

1 . DoD invests in commercial standards developments 

2. Change FAR/DFAR and SD-1 as appropriate 

B. REVISE THE MILESTONES FOR SOFTWARE-INTENSIVE 
DEVELOPMENTS (ACTION OFFICE USD(A&T)/ASD(C3I)) 

1 . Address the need for a software-first philosophy 

2. Provide for a layered software/hardware standards based architecture 


! 



C. DEVELOP DOD "BETA TEST" PHILOSOPHY (ACTION OFFICE OSD(T&E) 

1. Team with Universities/other appropriate activities 

2. Allow fielding of software direct from test beds with user consent 

IV. PERSONNEL 

A. DEVELOP SOFTWARE ACQUISITION MANAGERS (ACTION OFFICE 

USD(A&T)) 

1 . Provide a career path for software managers 

2. Integrate software personnel into senior acquisition staff 

B. PROVIDE SOFTWARE EDUCATION FOR SENIOR MANAGERS 
(ACTION OFFICE USD(A&T)) 

1 . Develop DoD Senior Software Managers Course 
.2. Develop and provide interactive training tools for 
senior managers to perfect software management skills 

C. PUBLISH AND PROMOTE "BEST PRACTICES" HANDBOOK (ACTION 
OFFICE OSD(T&E)) 

D. ENSURE DOMAIN EXPERTISE (ACTION OFFICE MILITARY 
DEPARTMENTS/ AGENCIES) 


V. SOFTWARE TECHNOLOGY BASE AND TRANSITION (ACTION OFFICE 
DDR&E) 

A. PROVIDE FOR SOFTWARE TECHNOLOGY INSERTION INTO SYSTEM 
ACQUISITION 

B. INVEST IN THE DOD SOFTWARE TECHNOLOGY STRATEGY 

C. PROVIDE FOR THE EVOLUTION OF THE DOD SOFTWARE 
TECHNOLOGY STRATEGY TO CAPTURE EMERGING COMMERCIAL 
PRACTICES 



DEPARTMENT OF THE ARMY 
OFFICE OF THE SECRETARY OF THE ARMY 
' 107 ARMY PENTAGON 
WASHINGTON DC 20310-0107 


Office, Director of Information 
Systems for Command, Control. 

Communications, & Computers 

MEMORANDUM FOR THE EXECUTIVE SECRETARY, DEFENSE SCIENCE BOARD 
TASK FORCE ON DOD ACQUISITION OF COMMERCIAL SOFTWARE 

SUBJECT: Proposed Action Plan with Regard to Common Changes Proposed by DoD 
Speakers to Revise the Software Acquisition Process 

The DSB Task Force asked us to provide a list of common recommended 
changes needed to revise DoD's software acquisition process. Our recommended list 
of proposed changes is provided in the enclosure to this memorandum. As we worked 
together to develop this list, we found it useful to categorize the recommended changes 
into five major categories. Several of our recommendations did not receive adequate 
emphasis in our presentations, yet are important to the process we are studying. We 
feel that if OSD is serious about addressing Software Acquisition issues, a single 
senior office with primary responsibility for effecting the reforms proposed by the DSB 
will be essential. Further, such an official or office must receive "from the heart" 
sustainment and backing from the highest levels within OSD. The enclosed list 
addresses our concerns with regards to the questions before the DSB. We hope that 
this will be beneficial to the DSB Task Force and we all look forward to continued active 


Lieutenant General, GS 
Director of Information (Communications, Computer, and Support 

Systems for Command, Control, Systems) 

Communications and Computers 




4 Feb 94 



participation in this effort. 





Date 


Printed on 


Recycled Paper 


Ref: 94-F-2088 


Mr. Jerome S. Gabig, Jr. 

Venable, Baetjer, Howard & Civiletti 
Suite 1000 

1201 New York Avenue, NW 
Washington, DC 20005-3917 

Dear Mr. Gabig: 


This responds to your September 22, 1994, Freedom of 
Information Act (FOIA) request pertaining to the Report of the 
Defense Science Board Task Force on Acquiring Defense Software 
Commercially. Our September 30 interim response refers. 

The Defense Science Board has provided the enclosed record 
as responsive to your request. There are no chargeable costs for 
processing this request, in this instance. 


Sincerely, 




A. H. Passarella 
Acting Director 
Freedom of Information 
and Security Review 


Enclosure: 

June 1994 DSB Report 

Prepared by Kahn : 4F2088L1 : 10/4/94 :DFOI :X71160 :gr_^pk 
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Executive Summary 


The Defense Science Board Task Force on Acquiring Defense Software Commercially recognizes 
that DoD systems are becoming increasingly dependent on the use of software as the mechanism for 
implementing operational capabilities. To adapt to changing military and national security situations, DoD 
is more dependent than ever on its ability to modify mission software rapidly, often in near real-time. 
However, software remains the schedule and cost driver for the development and maintenance of many 
important defense systems. 

In its review of current DoD and commercial software acquisition practices, the Task Force found 
notable differences, as evidenced in Appendix C. There are, however, many similarities between the 
various categories of DoD and commercial software systems. Although there are indications that 
commercial development efforts have achieved better predictability and lower costs, the Task Force noted a 
significant lack of credible, quantitative data to substantiate this assessment. 

In general, the Task Force concluded that DoD's investment in software requires greater DoD-wide 
management control and oversight in the coming years if the Department is to exploit the use of 
commercial software acquisition practices fully, as well as rapid advances in software technology. The 
following is a summary of selected findings and recommendations toward that end. 

Process Credibility: Current DoD practice is not compatible with commercial business 
practices. DoD should work to make necessary changes to acquisition regulations such as: 

• Having program managers manage 3 of 3 (price/schedule/functionality) but only constrain 2 of 3 

• Defining successful performance on contracts as delivering a solution (with predictable price, 
schedule and functionality) not adherence to government processes, procedures and specifications 

• Not requiring c-level specifications for software projects developed in Ada 

• Establishing mechanisms to allow both current ability to perform as well as past performance as 
key factors in source selection 

• Encouraging offerors to demonstrate as much functionality as possible as part of bid without 
eliminating domain knowledgeable competition 

DoD Program Management: DoD program management approaches discourage the use of 

commercial practices. Program managers lack incentives to tailor procedures to fit individual program 
needs or to develop “corporate” solutions (e.g., employ common architecture or common software 
components). DoD should establish and implement overarching software life cycle guidelines more 
conducive to the use of commercial practices and products, such as: 

• Defining software architectures to enable rapid changes and reuse 

• Facilitating early system engineering and iterative development 

• Participating in development of commercial and international standards 

• Allowing the fielding of software directly from test beds with user consent 

• Requiring program managers to stay with programs at least through beta testing to maintain 
continuity of understanding of original nuances in requirements 



